Virtual machine (vm) migration from switched fabric based computing system to external systems

ABSTRACT

An aspect of the present disclosure provides for migration of a virtual machine (VM) on a switched-fabric based computing system to another (external) switched-fabric based computing system or non-switched-fabric based (external) computing system. Both of such external computing systems are communicatively coupled to the switched-fabric based computing system by Internet such that the data transfer after the migration is based on IP packets on the Internet. According to another aspect of the present disclosure, an appliance contains multiple nodes for hosting virtual machines. A switched fabric switches packets between the nodes based on point-to-point connections between each pair of ports, wherein each port is coupled directly to a corresponding node. A bridge supports migration of a first VM to a first external system coupled to the first appliance by Internet Protocol (IP) based network.

BACKGROUND OF THE DISCLOSURE

1. Technical Field

The present disclosure relates to cloud computing, and more specificallyto virtual machine (VM) migration from switched fabric based computingsystem to external systems.

2. Related Art

Computing systems are often implemented to contain nodes interconnectedby switched fabrics. A node may contain computational and/or storageresources. A switched fabric relies on point-to-point connection betweeneach pair of input and output ports, for providing the switchingoperation.

Thus, in a common configuration, each node is coupled to one of theports of the fabric, and the fabric couples one port to another by acorresponding point-to-point connection, as required for transfer ofpackets between the corresponding nodes. The switched fabric may beimplemented using structures such as cross-bars, banyan switches, etcInfiniBand (IB) technology based computing system is an example of sucha computing system.

Virtual machines (VMs) may be provided within each of the computingnodes in an IB fabric. A VM is a self-contained operating environmentthat operates as if it were a separate machine, i.e., a separatecomputer. Virtual machines may be hosted in the computing nodes of an IBfabric based computing system to provide multiple instances of the sameor different operating system(s) for execution of respective sets ofapplications.

It is often necessary to migrate VMs (e.g., for efficient allocation ofcomputing resources) from one computing node to another. Migrationrefers to movement of a VM from a source computing node to a targetcomputing node for reasons such as load balancing, etc., while providingcontinued computational/storage resources and data connectivity(post-migration) to/from applications already executing in the VM afterthe migration.

Migration typically entails hosting the target computer node with animage of the migrating VM, transporting various state-information ofapplications and other software components to the operational image, andceasing the execution of the VM on the source computer node. Often suchmigration is within the same switched fabric based computing system inthat both the source and target nodes are contained in the computingsystem.

However, it may be desirable to migrate VMs to external systems also forreasons such as scalability, ability to meet requirements during peakdemand, cost-efficiencies, etc. Aspects of the present disclosureprovide VM migration from switched fabric based computing system toexternal systems, as described below with examples.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments of the present disclosure will be described withreference to the accompanying drawings briefly described below.

FIG. 1 is a block diagram illustrating an example environment in whichseveral aspects of the present disclosure can be implemented.

FIG. 2 is a flow chart illustrating the manner in which virtual machinesare migrated, according to an aspect of the present disclosure.

FIG. 3 is a block diagram illustrating the detailed architecture of someof the components of a switched fabric based computing system, in anembodiment.

FIG. 4A depicts VMs hosted in various computing nodes at one timeinstance.

FIG. 4B depicts VMs hosted in computing nodes after migration of some ofthe VMs at another time instance.

FIG. 5 is a flow chart illustrating the manner in which data istransferred to migrated virtual machines in an embodiment.

FIG. 6A depicts the content of a forwarding table prior to VM migration,in an embodiment of the present disclosure.

FIG. 6B depicts the content of a forwarding table after VM migration, inan embodiment of the present disclosure.

FIG. 7A depicts the content of a VTEP table prior to VM migration, in anembodiment of the present disclosure.

FIG. 7B depicts the content of a VTEP table after VM migration, in anembodiment of the present disclosure.

FIG. 8A depicts the content of a bridge table prior to VM migration, inan embodiment of the present disclosure.

FIG. 8B depicts the content of a bridge table after VM migration, in anembodiment of the present disclosure.

FIG. 8C depicts the content of a forwarding table in an external systemafter VM migration, in an embodiment of the present disclosure.

FIG. 8D depicts the content of a VTEP table in an external system afterVM migration, in an embodiment of the present disclosure.

FIG. 8E depicts the content of a bridge table in an external systemafter VM migration, in an embodiment of the present disclosure.

FIG. 9A illustrates the manner in which a MAC frame is encapsulated byan IB header to form an IB packet, in an embodiment of the presentdisclosure.

FIG. 9B illustrates the manner in which packets directed to migrated VMare transported from the switched fabric system initially hosting theVM, to the external system hosting the migrated VM, in an embodiment ofthe present disclosure.

FIG. 9C illustrates the manner in which packets originating frommigrated VM are transported to the switched fabric system initiallyhosting the VM, in an embodiment of the present disclosure.

In the drawings, like reference numbers generally indicate identical,functionally similar, and/or structurally similar elements. The drawingin which an element first appears is indicated by the leftmost digit(s)in the corresponding reference number.

DETAILED DESCRIPTION OF THE EMBODIMENTS OF THE DISCLOSURE 1. Overview

An aspect of the present disclosure provides for migration of a virtualmachine (VM) on a switched-fabric based computing system to another(external) switched-fabric based computing system or non-switched-fabricbased (external) computing system, as suitable in the correspondingduration. As a result, the types of external systems available formigrating VMs (from a switched fabric based computing systems) may beenhanced.

In an embodiment, in case of packets being transported to an externalsystem, the packets initially are encapsulated by fabric headersfacilitating internal switching based on the fabric headers. However,the fabric header is removed and encapsulated with another outer headersuitable for sending to both of switched-fabric based computing systemsand non-switched-fabric based computing systems.

According to another aspect, both of such external computing systems arecommunicatively coupled to the switched-fabric based computing system byInternet Protocol (IP) such that the data transfer after the migrationis based on IP packets. As a result, the number of external systemsavailable for migrating VMs may be enhanced.

Accordingly, in an embodiment, an appliance contains multiple nodes forhosting virtual machines. A switched fabric switches packets between thenodes based on point-to-point connections between each pair of ports,wherein each port is coupled directly to a corresponding node. A bridgesupports migration of a first VM to a first external system coupled tothe first appliance by Internet Protocol (IP) based network. Packets aretransported between the bridge and the external system based on IP.

Several aspects of the present disclosure are described below withreference to examples for illustration. However, one skilled in therelevant art will recognize that the disclosure can be practiced withoutone or more of the specific details or with other methods, components,materials and so forth. In other instances, well-known structures,materials, or operations are not shown in detail to avoid obscuring thefeatures of the disclosure. Furthermore, the features/aspects describedcan be practiced in various combinations, though only some of thecombinations are described herein for conciseness.

2. Example Environment

FIG. 1 is a block diagram illustrating an example environment in whichseveral aspects of the present disclosure can be implemented. The blockdiagram is shown containing appliances 110 and 130, remote farm 140,gateways 150-170, VM management system 190, and Internet 120. Merely forillustration, only representative number/type of systems are shown inthe Figure. Many environments often contain many more systems, both innumber and type, depending on the purpose for which the environment isdesigned. Each system/device of FIG. 1 is described below in furtherdetail.

Broadly, remote farm 140 represents a server farm housed in one or moreassociated data centers. Though shown as individual systems, appliances110 and 130 may be part of respective server farms, or contained withina same server farm connected by IP protocol, as described below. Inembodiments described below, each of appliances 110/130 and remote farm140 is ‘owned/operated’ by a corresponding business entity and aspectsof the present disclosure facilitate a VM in appliance 110 to bemigrated to appliance 130 or remote farm 140 and vice versa.

Remote farm 140 represents a non-switched-fabric based computing systemsuch as an Ethernet-based local area network, which rely on broadcastmedium for providing connectivity. Remote farm 140 is shown containingvarious computing nodes 145A-145N, storage nodes 148A-148M, localnetwork 146 and a bridge 147. Each component with a similar name (e.g.,bridge, CN, SN) performs similar function as the corresponding componentin appliance 110, and the description is not repeated for conciseness.

Appliance 130 is shown containing computing nodes 135A-135N, storagenodes 138A-138M, a switched fabric 136, and a bridge 137. Each of thesecomponents is connected in a similar fashion as the components inappliance 110, and performs similar functions as the components inappliance 110. Accordingly, the description of the components ofappliance 130 is not repeated for conciseness.

Internet 120 facilitates the connectivity of appliance 110 with externalsystems such as appliance 130 and remote farm 140. Internet 120 may beimplemented using protocols such as Transmission Control Protocol (TCP)and/or Internet Protocol (IP), well known in the relevant arts. Ingeneral, in TCP/IP environments, an IP datagram/packet is used as abasic unit of transport, with the source address being set to the IPaddress assigned to the source system from which the datagram originatesand the destination address set to the IP address of the destinationsystem to which the datagram is to be eventually delivered.

A (IP) datagram is said to be directed to a destination system when thedestination IP address of the datagram is set to the (IP) address of thedestination system, such that the datagram is eventually delivered tothe destination system by Internet 120. When the datagram containscontent such as port numbers, which specifies the destinationapplication, the packet may be said to be directed to such applicationas well. The destination system may be required to keep thecorresponding port numbers available/open, and process the datagramswith the corresponding destination ports. Internet 120 may beimplemented using any combination of wire-based or wireless mediums.

Each of the gateways 150, 160, and 170 represents a router that routeseach IP datagram from a source IP system to a destination IP system viaInternet 120, based on the IP address in the IP destination field.Specifically, a gateway processes an incoming datagram by examining thedatagram for a destination IP address, and forwards the datagram towardsthat destination system based on the IP address and routing information.Paths 111-114 and 121-123 represent the communication paths on which theIP datagrams are transported.

Appliance 110 is a switched fabric based computing system. Appliance 110is shown containing computing nodes 105A-105N, storage nodes 108A-108M,a switched fabric 106, and a bridge 107. Each of these components isdescribed below in further detail.

Each of storage nodes 108A-108M represents a non-volatile (persistent)storage component, facilitating storage and retrieval of data byapplications executing in computing nodes 105A-105N. For example,storage nodes 108A-108M may be implemented to store data, which would beof interest to end-users interfacing with applications on computingnodes 105A-105N. Accordingly, storage nodes 108A-108M may be implementedas a database server using relational database technologies or as anetwork attached storage (NAS) or a file server, providing storage andretrieval of data by applications, as is well known in the relevantarts.

Computing nodes 105A-105N represent processing entities that may hostone or more VMs, while potentially operating on the data stored instorage systems such as storage nodes 108A-108M. Computing nodes105A-105N provide computational capabilities within appliance 110, andeach computing node may be implemented as a blade server or a rackserver, available from various vendors. Each virtual machine may have anassociated layer-2 (e.g., Ethernet) and layer-3 (e.g., Internetprotocol) address, which are unique in their respective operatingnetwork domains (e.g., VLAN technology or Internetworking technology).

Switched fabric 106 provides connectivity between any two nodes ofcomputing nodes 105A-105N, storage nodes 108A-108M, and bridge 107 basedon point-to-point connection (contrasted with broadcast medium of802.3-type local networks). Each of the computing nodes 105A-105N,storage nodes 108A-108M, and bridge 107 is connected directly (insteadof by switching on networks, etc.) to a corresponding port of switchedfabric 106 by a physical link (shown as 101A-N, 102A-M, and 103Arespectively) that offers bi-directional data communication between thenodes and switched fabric 106. Each physical link can contain multiplelines so that several bits can be transferred in parallel, includingduring the switching operation. Switched fabric 106 forwards incomingdata from a source node to the port corresponding to a destination nodebased on identifiers of virtual machines or the bridge, to operate as aswitch.

Bridge 107, provided according to an aspect of the present disclosure,supports virtual machines on appliance 110 to be migrated to externalsystems such as appliance 130 or remote farm 140 (as suited in thecorresponding situation), and also enables virtual machines fromexternal systems to be migrated to appliance 110, as described insections below with examples.

VM management system 190 determines when and where to migrate VMsexecuting in several switched fabric computing systems (such asappliance 110) of an enterprise. Though only a single appliance 110 isshown as being served by VM management system 190 of the enterprise,several more of such appliances may be located within the enterprise andserved by VM management system 190. For the purpose of furtherdescription below, it is assumed that appliance 130 and remote farm 140are present external to such an enterprise. Though not shown, appliance130 may have a corresponding VM management system in the enterprise inwhich appliance 130 is placed.

VM management system 190 may be implemented as software on a server,which can be part of the server farm housed in one or more associateddata centers as described above. Using VM management system 190, anadministrator can also manually initiate the migration of a VM to otherappliances within the enterprise, or to an external system (i.e.,appliances 130 or remote farm 140).

Assuming a VM is to be migrated to another appliance within thesame/native enterprise, VM management system 190 thereafter operates inconjunction with the source and destination appliances to manage themigration. Such migration may be performed in a known way.

However there may be situations it may be desirable to migrate a VM toexternal enterprises, or to migrate VMs from external enterprises intoappliance 110. The manner in which VMs in the enterprise may be migratedto external machines is described below with examples.

3. Migration of Virtual Machines

FIG. 2 is a flow chart illustrating the manner in which VMs aremigrated, according to an aspect of the present disclosure. Theflowchart is described with respect to the systems of FIG. 1 merely forillustration. However, the features can be implemented in other systemsand environments also without departing from the scope and spirit ofvarious aspects of the present disclosure, as will be apparent to oneskilled in the relevant arts by reading the disclosure provided herein.

In addition, some of the steps may be performed in a different sequencethan that depicted below, as suited to the specific environment, as willbe apparent to one skilled in the relevant arts. Many of suchimplementations are contemplated to be covered by several aspects of thepresent disclosure. The flow chart begins in step 201, in which controlimmediately passes to step 210.

In step 210, each computing node of appliance 110 hosts a correspondingset of VMs. Some of the VMs may be created on one node of appliance 110and migrated successively to one or more of other nodes of appliance130. A VM is said to be ‘native’ to a node/appliance at which the VM isfirst created/formed in operational state. Once the VM is moved to anexternal system (for example, as described below), the VM is said to bea ‘migrated’ VM in that external system. The system at which the VM wascreated as a native VM (and then migrated), is said to be a sourcesystem for that migrated VM. In general, the VMs hosted on a computingnode execute while sharing the physical processing resources (processingunits, random access memory, etc.) provided on the computing node.

In step 220, appliance 110 identifies a condition that requiresmigration of a VM from appliance 110 to an external system. Thecondition can be based on any parameters of interest, such as priorprocessing load on the various nodes in the appliances (including 110)in the enterprise, time of day/week/month, overhead of retaining a VM inappliance 110 versus migrating to an external system, etc. Theconditions can be specified by an administrator in the form of rules inVM management system 190, and appliance 110 may thereafter determine theexistence or occurrence of such a condition potentially in coordinationwith VM management system 190. Alternatively, a user may issue manualcommands for the migration by utilizing the management softwareimplemented on VM management system 190, on an as-needed basis. Ingeneral, it may be assumed that there are multiple such conditions, andone of such conditions is identified to have occurred in step 220.

In step 240, appliance 110 determines an external bridge contained in anexternal system that is suitable for hosting the virtual machine soughtto be migrated, in cooperation with VM management system 190. Theexternal bridge is accessible by an external IP address, and theexternal bridge communicates with a node of the external system that issuitable for hosting the virtual machine. The external system may bedetermined, for example, dynamically based on conditions on externalsystems, or based on administrator specified rules.

In an illustrative scenario, an owner/operator of appliance 110 may havecontracted with the owner of external systems to host (migrated) VMs,and one of such external systems may be conveniently selected (based onrules, potentially) for hosting the migrated VM. As described below withexample, the external system can be either switched-fabric basedcomputing system or non-switched-fabric based (external) computingsystem, as suitably determined at the corresponding duration.

In step 250, appliance 110 causes the VM identified in step 220 to bemoved to the external system, in cooperation with VM management system190. Such moving may entail configuring a computer node of the externalsystem with the various states (e.g., reflecting the states of variousapplications executing on the VM) internal to the migrating VM, andceasing the execution of the VM on the computer node previously hostingthe VM. Such tasks may be performed in a known way in cooperation withcomplementing implementation on the external system (and/or othersystems in the farm housing the external system). For example, an imageof the same type of VM may be used as a starting point on the externalsystem, and configuring the image with the current state of the VMsought to be migrated, and then making the configured image operational.

In step 270, data is transported from and to the migrated VM using theexternal IP address of step 240. It should be appreciated that the datasought to be transported correspond to inter-VM packets designed toprovide connectivity to applications on migrated VM (for example, as theapplications access data via storage nodes) or when applicationsexecuting on native VMs of appliance 110 continue to communicate withapplications executing on the migrated VM. In particular, data istransported in the form of IP datagrams having the external IP addressas the destination address.

Continuity of communication to and from the migrated VM is supported byoperation of step 270 (such that continuity of operation is provided forall applications, both internal to the migrated VM, and those fromexternal systems/nodes communicating with the applications executing onthe migrated VM). Thus, migration may be viewed as the combination ofsteps 250 and 270 together. The flow chart ends in step 299.

Due to the migration of VMs from appliance 110, more scalability isprovided for applications executing on appliance 110. In addition, dueto the migration of VMs from external systems to appliance 110, anyexcess computational capacity in appliance 110 may be used for hostingexternal VMs. Further, in view of using IP packets for transporting theinter-VM data, aspects of the present disclosure can be used to takeadvantage of the processing power of many external systems accessiblevia IP.

The features described above with respect to FIG. 2 can be implementedin various embodiments, as will be apparent to one skilled in therelevant arts by reading the disclosure provided herein. The descriptionis continued illustrating the detailed architecture of appliance 110, inone embodiment.

4. Detailed Architecture

FIG. 3 is a block diagram illustrating the detailed architecture ofswitched fabric 106, bridge 107, and computing node 105A of appliance110 in one embodiment. Computing node 105A is shown containing VMs391-393, and virtual network interface cards (vNIC) 399A-399C. Switchedfabric 106 is shown containing crossbar 370, fabric controller 310,subnet manager 320, migration list 330, and forwarding table 340. VMmanagement system 190 is shown with paths 191, 192A and 192B. Bridge 107is shown containing network manager 380, VTEP table 350, and bridgetable 360. Each of the blocks is described below in further detail.

Each VM 391-393 is shown communicating via a respective vNIC of vNICs399A-399C (on a respective path of paths 394A-394C), and may thus have amedium access control (MAC) address and IP address. As described below,the MAC address is retained even after migration, and used as the basisfor migrating the corresponding VM to external systems. Each VM isassumed to be connected to a corresponding port of switched fabric 106,and identified by a respective machine identifier (MID), which is aunique number within the corresponding appliance.

A MID may be represented by a global ID (GID) and/or a local ID (LID)depending on the particular implementation of the VMs in the IB fabricof appliance 110. To illustrate, the IB fabric of appliance 110 may beimplemented using one or more IB subnets. An IB subnet is a group ofports available in an IB fabric, where the combination of such one ormore IB subnets defines all the available ports in that particular IBfabric. In one implementation, if the number of desired VMs in appliance110 is below the number of ports available in one IB subnet, all the VMsmay be identified by respective LIDs. If, however, the number of desiredVMs in appliance 110 is more than the number of ports available in oneIB subnet, the VMs may be identified by respective GIDs, in addition tothe LIDs.

For the purpose of the present disclosure it may be understood that eachVM can be uniquely addressed by a corresponding MID, which can be eithera GID or LID, as suited in the corresponding environment. In thedescription below, it is assumed that each MID corresponds to GID, inview of the large number of VMs that are hosted in some commonenvironments.

Though not shown, each VM is assumed to be executing a corresponding setof user applications. In operation, each VM generates a base payload forsending to another VM. The corresponding vNIC thereafter encapsulatesthe base payload with a MAC header to generate a MAC frame. In addition,the MAC frame is further encapsulated with a fabric header suitable forswitching (and also higher protocol layer functions). For the purpose offorwarding of the payload, it may be sufficient to appreciate that thefabric header contains MID. In case of Infiniband based switch fabric,the fabric header and the packet format is described in a documententitled, “Infiniband Architecture Specification, Volume 1”, availablefrom InfiniBand Trade Association Administration, 3855 SW 153rd Drive,Beaverton, Oreg. 97006 (Ph. No.: 503.619.0565), which document isincorporated in its entirety herewith.

Crossbar 370 provides point-to-point connection between various ports(101A-N, 102A-M, and 103A of FIG. 1) of switched fabric 106. Althoughcrossbar 370 is shown implementing cross-bar architecture, any otherarchitecture (e.g., banyan switches) providing point-to-point connectionbetween various ports of switched fabric 106 may be employed. For easeof description, each bi-directional link to the port is shown separatelyas input path and output path. Thus CN1 in input paths 371 along withCN1 in output paths 372 represents the port corresponding to 101A ofFIG. 1.

Crossbar 370 connects each input path to a specific output path (if adata packet is to be transferred) according to control signals receivedfrom fabric controller 310. Multiple point-to-point connection paths maybe provided simultaneously, with each connection path between acorresponding pair of input/output ports for higher switchingperformance. In addition, each such connection path may be implementedas a bus to transfer multiple bits in parallel between the connectedinput and output port pair. Crossbar 370 may be implemented using any ofvarious available technologies, including the IB technology noted above.

Forwarding table 340 contains connection data indicating the specificport on which each VM and bridge 107 is currently accessible. Bridge 107and each VM of appliance 110 may be uniquely identified by an identifier(i.e., MID) and accordingly connection data may contain a mapping of theMID to the port number on which the corresponding VM/bridge areaccessible.

Fabric controller 310 controls the switching operation of crossbar 370by issuing the appropriate control signals on path 317. Fabriccontroller 310 receives data signals on path 303 indicating the MID ofsource VM (from which packet is being received) and the MID ofdestination VM (to which packet is to be sent). Fabric controllerswitches the packets to the appropriate output port based on data storedin forwarding table 340.

As may be readily appreciated, packets destined to local VMs areswitched to corresponding output port (to which the node hosting thedestination VM is coupled to). On the other hand, packets fromin-migrated VMs and those destined for out-migrated VMs are switched tonetwork manager 380 via 301. The control signals are issued on path 317to connect each input path with the corresponding output path determinedbased on forwarding table 340.

Appliance 110 may implement one subnet manager for each switched fabric(as represented by block 320 being contained in switched fabric 106).Alternately, if appliance 110 contains multiple switched fabrics, asingle subnet manager may be implemented for all such switched fabricstogether.

VM management system 190 examines various data and identifies VMs thatrequire migration from appliance 110 to an external system, based onexistence or occurrence of condition(s) previously specified by anadministrator. It then adds identifiers of such VMs to the migrationlist 330 through path 191. VM management system 190 may also add themigrated VM's IP addresses to the migration list 330 along with theidentifiers of such VMs (i.e., MIDs).

Subsequently, when the identified VMs are moved from appliance 110 to anexternal system (as described with reference to step 250 of FIG. 2),subnet manager 320 receives a link-down event from VM management system190 for the corresponding VMs (indicating that the moved VM is no longeraccessible at the port provided by the forwarding table 340).

For each VM for which a link-down event is received, subnet manager 320identifies the corresponding MIDs of the VMs from migration list 330,and updates the corresponding entry in forwarding table 340 to indicatethat the VM is reachable at (in effect, via) bridge 107. Such updatingimplies that the packets originating in any of the local VMs destined tomigrated VM would be forwarded to network manager 380 (at 301). In anembodiment, during migration, original VM image continues to beoperational until new VM image is fully ready to start. When it is, theoriginal VM is shutdown and the new one is started. This implies thatduring migration, the VMs do not lose ability to receive data for a longduration.

Alternatively, for each VM for which a link-down event is received,subnet manager 320 identifies the corresponding MIDs of the VMs frommigration list 330, and deletes the forwarding table's (340) entry forthat MID. Subnet manager 320 then uses the VM's IP address present inthe migration list 330 to build and transmit a gratuitous ARP to theVM's subnet that maps the VM's IP address to the bridge's MID. The VMsthat are in the same broadcast domain as the migrated VM update themapping of the migrated VM's IP address to the bridge's MID when theyreceive the Gratuitous ARP. Thus all further packets destined to themigrated VM will be sent to bridge 107.

Migration list 330 contains identifiers of VMs that are determined to besuitable for migration from appliance 110 to external systems. VMmanagement system 190 writes the MIDs of the to-be-migrated VMs inmigration list 330. Similarly, in the event of manual commands beingissued by an administrator using the VM management system 190 for themigration of VMs, the MIDs of such VMs is also written to migration list330 by VM management system 190. Once migration is complete (includingsetup of all the pertinent data tables), the corresponding entries maybe removed from migration list 330.

VM management system 190 facilitates the migration of VMs in and out ofappliance 110, and also writes entries that are stored in VTEP table 350and bridge table 360. VM management system 190 may communicate with thecounterpart equivalent component of the external system, and determinethe end point at the external system that supports the post-migrationdata transfer (similar to bridge 107) the migrated virtual machine(e.g., the IP address of the remote bridge at which the out-migrated VMis hosted). VM management system 190 writes the correspondinginformation in VTEP table 350. In particular, VM management system 190updates VTEP table 350 to map each MAC address of a corresponding VM tothe parameters which specify the end point (i.e., bridge 107 orcounterpart in other systems) which supports packet transfers from andto the migrated VMs.

With respect to in-migration of VMs created on external systems, VMmanagement system 190 may cooperate with a counterpart component todetermine the end points at the external system and the appliance. Inaddition, VM management system 190 fetches the unique MID for the VMthat was assigned by subnet manager 320, and updates bridge table 360 toindicate assignment of the created MID for the in-migrated VM. SubnetManager 320 updates forwarding table 340 to reflect the specific nodehosting the in-migrated VM. In particular, the created MID is mapped tothe port connecting to the node hosting the in-migrated VM.

Each of the entries in the VTEP table 350 specify end pointscorresponding to remote VMs of interest, including VMs in appliance 110that have been migrated out (“out-migrated VMs”) of appliance 110, andthe remote VMs with which the in-migrated VMs communicate. In anembodiment, VXLAN technology is used for connecting appliance 110 withappliance 130 and remote farm 140, and accordingly for each in-migratedand out-migrated VM, the parameters stored in the corresponding VTEPtable 350 entry include an IP address of the bridge which serves the VM,the MAC address of the migrated/remote VM, and a virtual networkidentifier (VNI) identifying a segment of VXLAN over which the migratedVMs communicate with external systems. Although the VTEP table is shownas a single table for all VNIs, in other implementations, a single VTEPtable may be provided for each VNI (VXLAN segment). The parameters aredescribed below in further detail with respect to FIGS. 7A and 7B.

Bridge table 360 maintains mapping of VNIs to respective VMs that havein-migrated to, or out-migrated from appliance 110. Specifically, foreach VM that has in-migrated to or out-migrated from appliance 110, anentry is stored in bridge table 360 that maps the VM's MAC address tothe corresponding MID (within appliance 110), IP address, and VNI.Although the bridge table is shown as a single table for all VNIs, inother implementations, a single bridge table may be provided for eachVNI. The content of bridge table 360 in the context of VXLAN technologyis described below with respect to FIGS. 8A and 8B.

Network manager 380 works with the data structures provided in VTEPtable 350 and bridge table 360 and facilitates continued dataconnectivity between migrated VMs and VMs that are native in appliancesfrom which the migrated VMs migrate from. By providing continued dataconnectivity, network manager 380 provides for transfer of data to andfrom the migrated VMs.

The description is continued below with some examples, illustrating themigration of VMs between appliance 110, appliance 130, and remote farm140.

5. Migration of VMs

FIGS. 4A and 4B together show the migration of VMs hosted withincomputing nodes of appliance 110, appliance 130, and remote farm 140 attime instances before and after the migration of VMs. Though only VMsare shown to be migrated, the corresponding vNICs are also transportedto the corresponding destination nodes (along with the migrated VMs).

The migration of VMs is shown with respect to certain computing nodes ofFIG. 1 merely for illustration. However, the features can be implementedin other computing nodes and with any number of VMs (from remote farm140 to appliance 110 and vice versa, from appliance 130 to appliance 110and vice versa) without departing from the scope and spirit of variousaspects of the present disclosure, as will be apparent to one skilled inthe relevant arts by reading the disclosure provided herein.

Referring to FIG. 4A (representing a first instance of time), appliance110 is shown containing computing nodes 105A, 105B, and 105N. Incomputing node 105A, three VMs (with GIDs 1, 2, and 3) are shown hosted.In computing node 105B, one VM (GID4) is shown hosted. In computing node105N, one VM (GID5) is shown hosted. Appliance 130 is shown containingone computing node 135A, hosting one VM (with GID 6). Remote farm 140 isshown containing one computing node 145A, hosting two VMs (with GIDs 7and 8). Although MIDs are shown as being implemented as GIDs for thepurposes of illustration, it will be apparent to one skilled in the artby reading the disclosure provided herein that MIDs can be extended tobe implemented as LIDs in case of smaller scale environments, withoutdeparting from the scope and spirit of the disclosure.

Referring to FIG. 4B (representing a second instance of time that occursafter the first instance of time), appliance 110 is shown containingcomputing nodes 105A, 105B, and 105N. In computing node 105A, three VMs(with GIDs 1, 3, and 7) are shown hosted, showing a migration of VM GID2away from appliance 110, and an in-migration of VM7 to appliance 110.There are no changes shown in computing nodes 105B and 105N. Appliance130 is shown with one computing node 135A, hosting two VMs (with GIDs 2and 6), showing the migration of VM GID2 from appliance 110 to appliance130. Remote farm 140 is shown with one computing node 145A, hosting oneVM (with GID 8), showing the migration of VM GID7 from remote farm 140to appliance 110.

The description is continued below with some examples, illustrating thetransfer of data between VMs on appliance 110 and appliance 130 usingVXLAN (Virtual Extensible LAN) technology.

6. Data Transfer Using VXLAN Technology

FIG. 5 is a flow chart illustrating the manner in which data istransferred from a switched fabric based computing system to an externalsystem using VXLAN technology, according to an aspect of the presentdisclosure. The description corresponds to only the case of transferringpackets from the native/local VM GID1 of node 105A to VM GID2, which wasshown to have migrated to node 135A in FIG. 4B. The packet formats forthe various other cases will be apparent to one skilled in the relevantarts by reading the disclosure provided herein, and is not repeated herefor conciseness. The flow chart begins in step 501, in which controlimmediately passes to step 510.

In step 510, vNIC 399A on appliance 110 sends a first IB packet to VMGID2 that has been migrated to the external system appliance 130. It isassumed that the data payload of the IB packet is received on path 394Afrom VM GID1. The first IB packet contains the data payload, a MACheader, and a first IB header. The data payload and the MAC headertogether may be viewed as a MAC frame. The MAC header specifies the MACaddresses for the source VM and the destination VM. The IB headerspecifies GID1 as the source MID and GID2 as the destination MID. Fabriccontroller 310 accordingly switches the packet to bridge 107 based onthe content of forwarding table 340.

In step 520, bridge 107 removes the first IB header from the first IBpacket, thereby recovering the original MAC frame.

In step 530, bridge 107 adds a VXLAN header, a UDP header, an IP header,and an outer Ethernet header to the MAC frame to form a VXLAN packet (asdescribed in further detail in RFC 7348 entitled, “Virtual eXtensibleLocal Area Network (VXLAN): A Framework for Overlaying Virtualized Layer2 Networks over Layer 3 Networks”).

Specifically, the outer Ethernet header contains the MAC address for thesource (i.e., bridge 107), MAC address for the first intermediatedestination in the path to reach VM GID2 (i.e., gateway 150), IPaddresses for the source (i.e., bridge 107) and destination (i.e.,bridge 137). All the fields added outside of the basic/original MACframe (including fields 910-935 in FIG. 9B, described below), may betermed as an outer header.

Bridge 107 determines the IP address of the destination bridge 137 basedon the corresponding entry in VTEP table 350 for VM GID2, whichspecifies the IP address of the destination bridge with reference to VMGID2. Bridge 137 is remote bridge that is connected to appliance 130 atwhich the out-migrated VM is hosted, and is therefore the end point ofthe VXLAN connection.

The VXLAN packet is then sent from the first router (gateway 150) to thesecond router (gateway 160). As the packet is sent from gateway 150 togateway 160, the MAC addresses (or layer-2 address, in general) for thesource and destination in the outer Ethernet header are updated toreflect the corresponding MAC addresses for gateway 150 and gateway 160.

In step 540, the VXLAN packet is transmitted to the second bridge 137 asthe end point of the VXLAN connection from the second router, gateway160. Second bridge 137 is part of the external system, i.e., appliance130, hosting the migrated VM GID2. Second bridge 137 is in communicationwith a corresponding fabric controller in appliance 130, which in turncommunicates with the destination VM GID2 through a correspondingcrossbar framework.

In step 550, bridge 137 removes the VXLAN header from the VXLAN packetto recover the original MAC frame.

In step 560, bridge 137 adds a second IB header to the MAC frame tocreate a second IB packet. The second 113 packet contains the originalMAC frame along with the second 113 header. It should be appreciatedthat the IB header facilitates the packet to be switched for theswitched fabric, and thus the MAC frame with the IB header may bereferred to as a ‘switched packet’.

In step 570, bridge 137 sends the second IB packet to a switched fabric136 connected to the second VM GID2. The corresponding fabric controllerin appliance 130 switches the packet to VM GID2 based on the content ofa corresponding forwarding table.

In step 580, the second 113 header is removed from the second IB packet(after completion of switching by the switch fabric 106) and theresulting MAC frame is transmitted to the second VM GID2. The flow chartends in step 599.

Although the examples above have been made with reference to a datapacket being sent from one switched fabric based computing system (110)to another switched fabric based computing system (130), aspects of thepresent disclosure can be extended with reference to a data packet beingsent from a switched fabric based computing system (110) to anon-switched-fabric based computing system (140), as will be apparent toa skilled practitioner.

In particular, in the event of the data payload arriving at thenon-switched-fabric based computing system encapsulated in a VXLANpacket, the data payload may be processed using corresponding technologyimplemented on the external system (140) for forwarding the original MACframe to the destination VM. For example, in remote farm 140 thatrepresents a non-switched-fabric based computing system such as anEthernet-based local area network, local network 146 may rely onEthernet broadcast to send the MAC frame of step 550 to the destinationVM, as would be readily understood by those skilled in the relevantarts.

It may thus be appreciated that by removing the IB header in step 520,the approach of FIG. 5 is able to interoperate with processing nodesimplemented in switched fabric environments as well asnon-switched-fabric environments. However, in case a destination VM islocated in a switched Infiniband fabric based appliances, a suitable IBheader is added in step 560, as described above.

The features described above with respect to FIG. 5 can be implementedin various embodiments, as will be apparent to one skilled in therelevant arts by reading the disclosure provided herein. The descriptionis continued below with some examples, illustrating changes incorresponding data structures before and after the migrations of VMsshown with reference to FIGS. 4A and 4B.

7. Data Structures

FIGS. 6A-8E illustrates the manner (logically) in which data ismaintained in relation to VM migration in one embodiment. As with theexamples in FIGS. 4A, 4B, and 5, although MIDs are shown in FIGS. 6A-8Eas being implemented as GIDs, it will be apparent to one skilled in theart by reading the disclosure provided herein that MIDs can be extendedto be implemented as LIDs in case of smaller scale environments, withoutdeparting from the spirit of the disclosure.

FIG. 6A shows the state of forwarding table 340 in appliance 110 at afirst time instance, prior to the migration of VMs, and FIG. 6B showsthe state of forwarding table 340 at a second time instance after themigration of VMs.

Referring to FIG. 6A, forwarding table 340 there depicts a portion ofconnections data maintained in switched fabric 106 prior to themigration of VMs shown in FIG. 4A. Column 610 (“MID”) specifies a GIDfor bridge 107 and for each VM in each computing node of appliance 110.Column 630 (“Port”) specifies the port number in switched fabric 106,which uniquely identifies the port to which the bridge connects to, orthe node hosting the corresponding VM connects to. Any data to theVM/bridge identified in column 610 is routed to the port numberidentified in column 630.

Each of rows 601-606 specifies the connection details of a particular VMor of bridge 107 represented in switched fabric 106. In particular, row601 specifies that the VM with a MID of GID1 is accessible via IB Port 1such that all data sent to VM GID1 will be forwarded by switched fabric106 to IB Port 1. Similarly other VMs (rows 602-605) with correspondingGIDs are shown with their corresponding connection ports. Additionally,row 606 specifies that bridge 107 with a MID of GID20 is accessible viaIB Port 10 such that all data sent to the bridge will be forwarded byswitched fabric 106 to IB Port 10.

It is also pertinent to note that VMs on the same computing node may beconfigured to be connected to the same port number in switched fabric106. For instance, GIDs 1, 2, and 3, previously shown in FIG. 3 ashaving been contained in the same computing node 105A, are all shown tobe associated with the same port number 1.

Referring to FIG. 6B, forwarding table 340 depicts a portion ofconnections data maintained in switched fabric 106 after migration ofVMs shown in FIG. 4B.

Row 602, which previously showed VM GID2 connected to IB Port 1, nowshows VM GID2 being connected to IB Port 10. IB Port 10 is the portnumber on switched fabric 106 that is connected to bridge 107 (as shownby row 606). As noted earlier, any VM (such as VM GID2) that migratesaway from any computing node within appliance 110 is assigned the portnumber of bridge 107, so that any data directed towards the migrated VMis thereafter sent to bridge 107 at port 10 for further processing.

Newly created row 607 contains the GID information for VM GID7. Aspreviously shown in FIG. 4B, VM GID7 migrated from remote farm 140 tocomputing node 105A of appliance 110. For the purpose of illustratingthat VM management system 190 assigns a new MID to the in-migrated VM,the GID of the in-migrated VM is shown as GID7A to represent the updatedMID assigned to the VM which was previously identified (prior tomigration) with a GID7. As with the other VMs on computing node 105A(i.e., GID1 and GID3), the newly migrated VM GID7A is also assigned thesame port (i.e., IB Port 1) on switched fabric 106. IB Port 1 nowaccepts all data directed towards the newly migrated VM GID7A. Thecontent of the rest of the rows (601, 603-606) remain unchanged from thepreviously shown content of forwarding table 340 in FIG. 6A.

FIG. 7A shows the state of VTEP table 350 in appliance 110 at a firsttime instance, prior to the migration of VMs, and FIG. 7B shows thestate of VTEP table 350 at a second time instance after the migration ofVMs.

Referring to FIG. 7A, VTEP table 350 depicts the VTEP table data ofremote VMs with which in-migrated VMs communicate, and data ofout-migrated VMs maintained in bridge 107 prior to the migration of VMs,as shown in FIG. 4A. Since there are no VM migrations at the timeinstance of FIG. 4A, VTEP table 350 is shown empty.

Referring to FIG. 7B, VTEP table 350 depicts the VTEP data of remote VMswith which in-migrated VMs communicate, and data of out-migrated VMsmaintained in bridge 107 after the migration of VMs to and fromappliance 110 as shown in FIG. 4B. Since one VM GID2 migrated away fromappliance 110 at the time instance corresponding to FIG. 4B, and sinceno in-migrated VM (e.g., VM GID7A) is shown communicating (e.g., to senddata) with remote VMs (e.g., VM GID8), VTEP table 350 is shown with oneentry. The entries in VTEP table 350 are updated by VM management system190, as previously described.

Column 710 (“VNI of the migrated VM”) specifies a VXLAN NetworkIdentifier (VNI) for the corresponding migrated or remote VM. As is wellknown in the relevant arts, each VNI uniquely identifies a correspondingVXLAN segment hosting several MAC nodes. Different VMs (even in the samecomputing node) can be part of different VXLAN segments, and thereforebe assigned different VNIs.

Column 720 (“Mac Address of the migrated VM”) specifies the MAC addressfor the migrated or remote VM. As noted earlier, each VM in appliance110, appliance 130, and remote farm 140 is assigned a MAC address at thetime of instantiating the VM in the corresponding native system, and theMAC address is retained by the VM both pre and post migration.

Column 730 (“IP Address of the bridge”) specifies an IP address of thebridge of the system that hosts the migrated or remote VM. A VTEP (VXLANTunnel Endpoint) represents an endpoint of a tunnel (implemented on UDP)for terminating the VXLAN packets. Since it is the bridge that acceptsany incoming VXLAN packets directed to the migrated or remote VM, the IPaddress of the bridge (hosting the VTEP) is stored in this column.

Row 701 specifies the address details of the corresponding out-migratedVM. In particular, row 701 specifies that the out-migrated VMcommunicates over the network using a VNI of 100, and has a MAC address“MAC2”. The IP address of the remote bridge 137 hosting the out-migratedVM is “IP-VTEP2”.

FIG. 8A shows the state of bridge table 360 in appliance 110 at a firsttime instance, prior to the migration of VMs, and FIG. 8B shows thestate of bridge table 360 at a second time instance after the migrationof VMs.

Referring to FIG. 8A, bridge table 360 depicts the bridge table data ofin-migrated and out-migrated VMs maintained in bridge 107 prior to themigration of VMs, as shown in FIG. 4A. Since there are no VM migrationsat the time instance corresponding to FIG. 4A, bridge table 360 is shownempty.

Referring to FIG. 8B, bridge table 360 depicts the bridge table data ofmigrated VMs maintained in bridge 107 after the migration of VMs to andfrom appliance 110 as shown in FIG. 4B. Since one VM GID2 migrated awayfrom appliance 110 and another VM GID7 migrated into appliance 110 atthe time instance corresponding to FIG. 4B, bridge table 360 is shownwith two entries. The entries in bridge table 360 are updated by VMmanagement system 190, as previously described.

Column 810 (“Mac Address of the migrated VM”) specifies the unique MACaddress for the migrated VMs. Since the MAC address is assigned at thetime of instantiation of the VM in the VMs native system, and since theunique MAC address is retained by the VM both pre and post migration,the MAC address shown in this column is the same as that carried by themigrated VMs prior to their migration.

Column 820 (“MID of the migrated VM locally”) specifies a MID for eachmigrated VM, which is assigned to the VM locally within the system wherethe VM migrates to.

Column 830 (“IP Address of the migrated VM”) specifies the IP address ofthe migrated VM, which like the VMs MAC address is assigned at the timeof instantiation of the VM in the VMs native system, and is retained bythe VM both pre and post migration.

Column 840 (“VNI of the migrated VM”) specifies VNI over which themigrated VM communicates.

Row 801 specifies the address details of the corresponding out-migratedVM GID2. In particular, row 801 specifies that the GID of theout-migrated VM is GID2 (same value as pre-migration), the MAC and IPaddresses of the out-migrated VM are MAC2 and IP2 respectively, and thatVM GID2 communicates over the network using a VNI of 100.

Row 802 specifies the address details of the corresponding in-migratedVM GID7A (a new value assigned to the VM by Subnet Manager 320, uponin-migration from remote farm 140). In particular, row 802 specifiesthat the GID of the in-migrated VM is GID7A, the MAC and IP addresses ofthe in-migrated VM are MAC7 and IP7 respectively, and that VM GID7Acommunicates over the network using a VNI of 200.

FIGS. 8C, 8D, and 8E illustrate the manner (logically) in whichforwarding, VTEP and bridge table data of migrated VMs is maintained inappliance 130, where VM GID2 has out-migrated from appliance 110. Eachcolumn of FIGS. 8C-8E has the same title/operation as the correspondingcolumn of FIGS. 6A, 7A, and 8A respectively, and the description is notrepeated for conciseness.

Referring to FIG. 8C, forwarding table 850 depicts the forwarding tabledata maintained in switched fabric 136 after migration of VMs intoappliance 130 shown in FIG. 4B.

Row 803 specifies that the native VM with a MID of GID6 is mapped to IBPort 11 (within appliance 130) such that all data addressed to VM GID6will be forwarded by switched fabric 136 to IB Port 11. Row 804specifies that bridge 137 with a MID of GID40 is accessible via IB Port20 such that all data sent to the bridge will be forwarded by switchedfabric 136 to IB Port 20. Row 805 contains GID and port information forthe in-migrated VM GID2A (a new value assigned to the VM by acorresponding subnet manager in appliance 130, upon in-migration fromappliance 110). Row 805 shows VM GID2A being connected to IB Port 11. Aswith the other VM on computing node 135A (i.e., GID6), the newlymigrated VM GID2A is also assigned the same port (i.e., IB Port 11) onswitched fabric 136. IB Port 11 now accepts all data directed towardsthe newly migrated VM GID2A.

Referring to FIG. 8D, VTEP table 860 depicts the VTEP table data of aremote VM maintained in bridge 137 after the in-migrated VM GID2Acommunicates with remote VM GID1. Row 806 specifies the address detailsof a remote VM GID1. In particular, row 806 specifies that the remote VMcommunicates over the network using a VNI of 100, and has a MAC address“MAC1”. The IP address of bridge 107 hosting the remote VM is“IP-VTEP1”.

Referring to FIG. 8E, bridge table 870 depicts the bridge table data ofmigrated VMs maintained in bridge 137 after the migration of VMs asshown in FIG. 4B. Since one VM (GID2) in-migrated to appliance 130 fromappliance 110 at the time instance corresponding to FIG. 4B, bridgetable 870 is shown with one entry. The entries in bridge table 870 areupdated by a corresponding VM management system in appliance 130.

Row 807 specifies the address details of the corresponding in-migratedVM GID2A. In particular, row 807 specifies that the GID of thein-migrated VM is GID2A, the MAC and IP addresses of the in-migrated VMare MAC2 and IP2 respectively, and that VM GID2A communicates over thenetwork using a VNI of 100.

The manner in which the data is transported using the data structuresthus maintained, is described below with examples.

8. Forwarding of Data

Network manager 380 facilitates transporting of data from and tomigrated VMs using the external IP address. The operation of networkmanager 380 in facilitating the transport of data is described belowwith reference to data sent from a native VM to an out-migrated VM andvice-versa.

Consider the case of data being sent from VM GID1 to VM GID2. As shownin FIGS. 4A and 4B, VM GID2 migrated from appliance 110 to appliance130. VM GID1 generates a base payload for sending to VM GID2. vNIC 399Athereafter encapsulates the base payload with a MAC header to generate aMAC frame. In addition, the MAC frame is further encapsulated with an IBheader suitable for switching, thus forming an IB packet.

As noted above, upon migration of VM GID2, forwarding table 340 inappliance 110 is updated such that any data directed towards VM GID2 issent to IB Port 10 (at which bridge 107 is connected to switched fabric106). Therefore, due to the corresponding configuration of forwardingtable 340, when switched fabric 106 receives the IB packet destined toVM GID2, the IB packet is sent to network manager 380 at bridge 107 forfurther processing.

Among other things, the IB packet contains the GID for the source VM andthe destination VM. Network manager 380 examines bridge table 360 toretrieve the VNI and MAC address corresponding to the migrated VM basedon the GID of the destination VM present in the received IB packet. Toillustrate, network manager 380 examines bridge table 360 shown in FIG.8B and finds a match for GID2 in row 801. It then retrieves the VNI(100) and the MAC address (MAC2) of VM GID2 based on the GID match.

Thereafter, based on the retrieved VNI entry and MAC address, networkmanager 380 examines the VTEP table 350 to retrieve the IP address ofthe remote bridge hosting the out-migrated VM GID2. To illustrate,network manager 380 examines bridge table 350 shown in FIG. 7B toretrieve the IP address IP-VTEP2 for the destination VM GID2 based onmatching the VMs VNI and MAC address as shown in row 701.

Network manager 380 strips the IB packet of the IB header and generatesa VXLAN packet based on the parameters of the entry (e.g., the IPaddress of the destination VTEP/bridge) found in the VTEP table 350.Specifically, once the IB header is stripped from the IB packet, theoriginal MAC frame is exposed for further processing. The original MACframe is encapsulated with a VXLAN header (that uses the VNI informationfound in VTEP table 350), a UDP header, an IP header (that uses the IPaddress of the destination bridge found in VTEP table 350), andoptionally, an outer Ethernet header that contains the MAC addresses ofthe source and destination VTEP end points. The VXLAN packet is thensent as a corresponding datagram on path 111.

Upon reaching the destination bridge 137, the VXLAN packet is examinedby a corresponding network manager on bridge 137. First, thecorresponding network manager examines the relevant bridge table entry,i.e., row 807 in bridge table 870 shown in FIG. 8E, based on the VNIpresent in the received VXLAN packet. Then, the corresponding networkmanager retrieves the GID of the destination VM based on the MAC addressof the destination VM specified in the VXLAN packet. The correspondingnetwork manger then strips the VXLAN packet to expose the base payloadand transmits the base payload to the destination VM (with the bridge asthe source) based on the corresponding entry in the forwarding table 850(i.e., row 805 in table 850 shown in FIG. 8C). The base payload isencapsulated in an IB frame during the transport of the payload frombridge 137 to VM GID2A. However, prior to the delivery of the basepayload to the destination VM, the IB and MAC frames are removed by acorresponding vNIC in communication with destination VM GID2A.

Consider the case of data being sent from VM GID2A to VM GID1. VM GID2Agenerates a base payload for sending to VM GID1. In an embodiment, whenVM GID2A migrates to appliance 130, all the internal routing tables ofVM GID2A are reset such that VM GID2A no longer has internal referencesof GIDs to VMs that are native to appliance 110.

After the re-set of internal routing tables, and prior to sending a datapacket for the first time, an ARP request is issued by VM GID2A todetermine the address parameters (MID and MAC address) of thedestination VM (i.e., the intended recipient of the data packet). TheARP request is processed by bridge 137, which creates a multicast IPpacket with the ARP request in it and forwards the multicast to allVTEPs in the VNI over which VM GID2A communicates.

Thereafter, bridge 107 receives the multicast packet and sends the ARPrequest to VM GID1. VM GID1 then creates a unicast ARP reply and sendsthe reply to bridge 107, which embeds the reply in a VxLAN packet andsends it to bridge 137.

Upon receiving the unicast packet, bridge 137 creates an entry in itsVTEP table (i.e., entry 806 in FIG. 8D) and stores the MAC address of VMGID1 and the IP address of the bridge 107 (IP-VTEP1) in the VTEP table.Bridge 137 then sends the MAC address of VM GID1 to VM GID2A as aresponse to the ARP request. The ARP response may further specify thatthe MID of bridge 137 (i.e., GID40) as being the MID of GID1 (eventhough VM GID1 has some other MID within appliance 110). In general, theMID of all VMs in native appliance 110 (from which VMs have migrated into appliance 130) is mapped to GID40 in appliance 130.

VM GID2A then sends a base payload (data) destined to VM GID1. A vNIC incommunication with VM GID2A thereafter encapsulates the base payloadwith a MAC header to generate a MAC frame. In addition, the MAC frame isfurther encapsulated with an IB header (with the local bridge MID ofGID40 as the fabric header destination) suitable for switching, thusforming an IB packet. The corresponding network manager on bridge 137strips the IB packet of the IB frame and encapsulates the exposed MACframe in a VXLAN packet using the VTEP and MAC address information inthe VTEP table of FIG. 8D. The encapsulation of the VXLAN packet isperformed in a similar fashion as described with reference to thetransport of data from VM GID1 to VM GID2 above.

Once the VXLAN packet is received at bridge 107, network manager 380examines the relevant bridge table entry, i.e., row 801 in bridge table360 shown in FIG. 8B, based on the destination MAC address of payloadframe and VNI present in the received VXLAN packet. If no matching entryis found, then network manager 380 examines an internal table (notshown) maintained on appliance 110 to determine the GID for thedestination VM (GID1) based on the MAC/IP destination addresses in thepayload of the VXLAN packet. The internal table can be populated basedon ARP type protocols and also examination of source MID of variouspackets received from fabric controller 310. For example, if theinternal table also does not have a matching entry, then an ARP requestfor that MAC address is sent by bridge 107 to a Multicast GID of the IBpartition mapped to that VNI. This is received only by VMs of thatpartition/tenant. The reply uniquely identifies the GID within that IBpartition/VNI. Network manager 380 then sends the packet using the GIDin the response as destination GID. In an alternate embodiment, an entryis created in the bridge table for such a VM GID so that further packetsover VXLAN can find the entry in the bridge table 360.

Upon determining the GIDs for destination VM, network manager 380removes the VXLAN header from the VXLAN packet to recover the originalMAC frame. Thereafter, network manager 380 adds an IB header to the MACframe to create an IB packet. The IB packet is sent to switched fabric106, which then sends the IB packet to VM GID1 through normal switchingoperation described above.

It may thus be appreciated that when a VM is migrated from one switchedfabric based appliance to another similar appliance, packets continue tobe transported to and from the other VMs in the native appliance,without the applications/VMs necessarily being aware of the migration.

It may be further appreciated that implementation of bridge 147 will beapparent to one skilled in the relevant arts at least based on thedescription of above. In particular, in case of packets being receivedfrom appliance 110, once bridge 147 has recovered the original MACframe, the destination MAC address can be used to deliver the packet tothe local computing node hosting the destination VM. In case of transferof packets from an in-migrated VM in remote farm 140 to VMs in nativeappliance 110, bridge 147 may operate as a proxy ARP (similar to bridge137) in answering the ARP requests directed to VMs in native appliance110, and thereafter forward the received base MAC packets as VXLANpackets in a known way.

Thus by stripping off the IB header of data packets before beingtransported on Internet 120, the approaches described above enable VMsin appliance 110 to be migrated to switched fabric based computingsystems (i.e., appliance 130 in the above example) ornon-switched-fabric based computing systems (140).

The description is continued with respect to format of the data packetswhen transported from a source VM to a destination VM is described belowwith examples.

9. Packet Formats

As described above with reference to FIG. 5 and FIGS. 6A-8E, theoriginal MAC frame is transported to the destination VM after differentstages of encapsulation. Specifically, the original MAC frame isinitially encapsulated in an IB header to form an IB packet. Thereafter,the IB header is removed and the MAC frame is encapsulated by VXLANheader prior to sending the resulting VXLAN packet to the destinationbridge. It may be understood that the original MAC frame usesEthernet-over-IB (EoIB) encapsulation and then VXLAN encapsulation. Itis also possible that the original frame is an IP packet that isencapsulated in an IB packet initially. This type of encapsulation isreferred to as Internet-Protocol-over-IB (IPoIB). Thereafter, the IBheader is removed and the IP packet is encapsulated by VXLAN headerprior to sending the resulting VXLAN to the destination bridge. In thedescription below it is assumed that EoIB and VXLAN are used.

FIG. 9A shows the MAC frame being encapsulated in an IB frame in anexemplary data transfer from VM GID1 to VM GID2. FIG. 9B shows the MACframe as part of a VXLAN encapsulated Ethernet packet (after the IBheader is removed), and the three hops involved in transporting theVXLAN packet from bridge 107 to bridge 137. FIG. 9C shows the reversepath of the VXLAN encapsulated Ethernet packet, as the packet travelsfrom bridge 137 to bridge 107.

FIG. 9A illustrates the packet format of an IB packet transported fromswitched fabric 106 to bridge 107 in appliance 110, in one embodiment.Only some of the fields of the packet, as relevant to an understandingof the transporting of packet, are depicted in the Figure, forconciseness. The remaining fields will be apparent based on theapplicable standards, etc., some of which are noted above.

The IB packet in FIG. 9A shows an IB header and the original MAC frame.The IB header contains a global route header 972 containing the sourceGID 974 and the destination GID 976. Additionally, local route header970, base transport header 978, and other headers 980 all form part ofthe IB header.

As relevant to understanding of the operation of the embodimentsdescribed above, global route header 972 contains the GIDs (source GID974 and destination GID 976) of the source VM and the destination VM.For further details as well as description of remaining fields(including other headers 980), the reader is referred to the documententitled, “Infiniband Architecture Specification, Volume 1”, availablefrom InfiniBand Trade Association Administration, 3855 SW 153rd Drive,Beaverton, Oreg. 97006 (Ph. No.: 503.619.0565), noted above. As may bereadily appreciated, bridge 107 operates to strip off IB (fabric) headercontaining fields 970, 972, 978 and 980, and transmits original MACframe containing fields 940, 945, 950, 955 and 960 further encapsulatedby VXLAN header in the example embodiments described above.

FIGS. 9B and 9C illustrate the packet format of a VXLAN encapsulatedEthernet packet transported from one computing system to anothercomputing system, in one embodiment. Only some of the fields of thepackets, as relevant to an understanding of the transporting of packets,are depicted in the Figures, for conciseness.

The VXLAN encapsulated Ethernet packets in FIGS. 9B and 9C show an outerEthernet header containing the MAC address of the source VTEP (field910) and its next IP destination (field 915), an IP header containingthe source IP address of the source VTEP (field 920) and the destinationIP address of the destination VTEP (field 925), a UDP header (field930), a VXLAN header containing a VXLAN Network Identifier (VNI) (field935), and the original MAC frame that is sent from a local VM destinedto a remote VM containing the MAC address of the source VM (field 940),the MAC address of the destination VM (field 945), the IP address of thesource VM (field 950), the IP address of the destination VM (field 955),and the original base payload (field 960).

As the VXLAN encapsulated Ethernet packet moves from a source VTEP to anext IP destination (“hop”), the outer Ethernet header changes toreflect the MAC addresses of the source and the next IP destination ofthe packet. All other values in the packet remain unchanged throughoutthe transport from one computing system to another computing system.

FIG. 9B shows the packet structure and content of a VXLAN encapsulatedEthernet packet as it is transported from appliance 110 to appliance130, subsequent to a MAC frame being sent from a local VM GID1 (inappliance 110) destined to a migrated VM GID2A (in appliance 130). Thevalues associated with fields 910-960 in the VXLAN encapsulated Ethernetpacket during the packet's three hops from the appliance 110 toappliance 130 are shown by corresponding field values 901-903.

Field values 901 shows the values in the VXLAN encapsulated Ethernetpacket during the packet's hop from bridge 107 (source VTEP) to gateway150 (next IP destination). Accordingly, the outer Ethernet header showsthe source MAC address as MAC-VTEP-1 (corresponding to the MAC addressof bridge 107) and the destination MAC address as MAC-GW-1(corresponding to the MAC address of gateway 150).

Field values 902 shows the values in the VXLAN encapsulated Ethernetpacket during the packet's hop from gateway 150 (source VTEP) to gateway160 (next IP destination). Accordingly, the outer Ethernet header showsthe source MAC address as MAC-GW-1 (corresponding to the MAC address ofgateway 150) and the destination MAC address as MAC-GW-2 (correspondingto the MAC address of gateway 160).

Field values 903 shows the values in the VXLAN encapsulated Ethernetpacket during the packet's hop from gateway 160 (source VTEP) to bridge137 (next IP destination). Accordingly, the outer Ethernet header showsthe source MAC address as MAC-GW-2 (corresponding to the MAC address ofgateway 160) and the destination MAC address as MAC-VTEP-2(corresponding to the MAC address of bridge 137).

Referring further to the field values, the IP address of the source VTEP(field 920) will be readily available to bridge 107 at the time ofpacket formation since bridge 107 is the source VTEP. The IP address ofthe destination VTEP (field 925) and the VNI (field 935) is obtainedfrom the VTEP table 350 shown FIG. 7B based on the GID of destinationVM.

FIG. 9C shows the packet structure of a VXLAN encapsulated Ethernetpacket as it is transported from appliance 130 to appliance 110,subsequent to a MAC frame being sent from a migrated VM GID2A (inappliance 130) destined to a local VM GID1 (in appliance 110). Thevalues associated with fields 910-960 in the VXLAN encapsulated Ethernetpacket during the packet's three hops from the appliance 130 toappliance 110 are shown by corresponding field values 904-906. As notedearlier, each field in FIG. 9C has the same label/operation as thecorresponding field of FIG. 9B, and the description is not repeated forconciseness.

Field values 904 shows the values in the VXLAN encapsulated Ethernetpacket during the packet's hop from bridge 137 (source VTEP) to gateway160 (next IP destination). Accordingly, the outer Ethernet header showsthe source MAC address as MAC-VTEP-2 (corresponding to the MAC addressof bridge 137) and the destination MAC address as MAC-GW-2(corresponding to the MAC address of gateway 160).

Field values 905 shows the values in the VXLAN encapsulated Ethernetpacket during the packet's hop from gateway 160 (source VTEP) to gateway150 (next IP destination). Accordingly, the outer Ethernet header showsthe source MAC address as MAC-GW-2 (corresponding to the MAC address ofgateway 160) and the destination MAC address as MAC-GW-1 (correspondingto the MAC address of gateway 150).

Field values 906 shows the values in the VXLAN encapsulated Ethernetpacket during the packet's hop from gateway 150 (source VTEP) to bridge107 (next IP destination). Accordingly, the outer Ethernet header showsthe source MAC address as MAC-GW-1 (corresponding to the MAC address ofgateway 150) and the destination MAC address as MAC-VTEP-1(corresponding to the MAC address of bridge 107).

Once the VXLAN encapsulated Ethernet packets are received at therespective destination bridges (i.e., bridge 137 with respect to FIG.9B, and bridge 107 with respect to bridge 9C), the VXLAN encapsulatedEthernet packets will be processed as described in the disclosure above(see FIG. 3 and FIG. 5), with the corresponding switched fabricforwarding the original MAC packet to the respective destination VMs.

In FIGS. 9B and 9C, it may be appreciated that each of 901-906 can beviewed as a corresponding packet, with the values in each packet shownagainst respective fields shown at the top.

It should be appreciated that the features described above can beimplemented in various embodiments as a desired combination of one ormore of hardware, executable modules, and firmware. The description iscontinued with respect to an embodiment in which various features areoperative when executable modules are executed.

10. Computer Readable Medium

Though not shown, various components of appliance 110 (including bridge107, nodes 105A-N, and VM management system 190), may be driven bysoftware instructions provided from a non-volatile storage media/medium.The instructions may be retrieved into random access memories (forsuperior performance) and executed by the processors to provide variousfeatures described above, including (one or more of) providing arun-time environment for supporting VMs, the steps of FIG. 2, theimplementation of bridge 107, etc.

The term “storage media/medium” as used herein refers to anynon-transitory media that store data and/or instructions that cause amachine to operate in a specific fashion. Such storage media maycomprise non-volatile media and/or volatile media. Non-volatile mediaincludes, for example, optical disks, magnetic disks, or solid-statedrives. Volatile media includes dynamic memory, such as RAM.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics. Transmissionmedia can also take the form of acoustic or light waves, such as thosegenerated during radio-wave and infra-red data communications.

11. Conclusion

While various embodiments of the present disclosure have been describedabove, it should be understood that they have been presented by way ofexample only, and not limitation. Thus, the breadth and scope of thepresent disclosure should not be limited by any of the above-describedexemplary embodiments, but should be defined only in accordance with thefollowing claims and their equivalents.

It should be understood that the figures and/or screen shots illustratedin the attachments highlighting the functionality and advantages of thepresent disclosure are presented for example purposes only. The presentdisclosure is sufficiently flexible and configurable, such that it maybe utilized in ways other than that shown in the accompanying figures.

Further, the purpose of the following Abstract is to enable the PatentOffice and the public generally, and especially the scientists,engineers and practitioners in the art who are not familiar with patentor legal terms or phraseology, to determine quickly from a cursoryinspection the nature and essence of the technical disclosure of theapplication. The Abstract is not intended to be limiting as to the scopeof the present disclosure in any way.

What is claimed is:
 1. A first appliance comprising: a plurality ofnodes for hosting virtual machines (VMs), said plurality of nodesincluding a first node hosting a first VM and a second VM; a switchedfabric switching packets between said plurality of nodes based onpoint-to-point connections between each pair of ports, wherein each portis coupled directly to a corresponding node of said plurality of nodes;and a bridge to support migration of said first VM to a first externalsystem external to said first appliance, said first external systembeing coupled to said first appliance by Internet Protocol (IP) basednetwork, said first external system being addressable by a first IPaddress, wherein said bridge forwards packets from said second VM insaid first appliance to said first VM migrated to said first externalsystem in the form of IP datagrams, each IP datagram having adestination IP field set to said first IP address.
 2. The firstappliance of claim 1, wherein each of said VMs executing in said firstappliance is assigned a corresponding VM identifier (MID), wherein saidswitched fabric switches each packet received on one port to anotherport based on said MID contained in the packet.
 3. The first applianceof claim 2, wherein each of said VMs executing in said first applianceis assigned a corresponding MAC (medium access control) address, whereinsaid MAC address is maintained to be the same for said first VM in bothof said first appliance and said first external system.
 4. The firstappliance of claim 3, wherein each VM generates MAC packets destined forremote VMs, wherein each MAC packet is encapsulated by a fabric headerto form a switched packet designed for switching by said switchedfabric, wherein said bridge is operable to support a virtual local areanetwork (VLAN) on top of said IP based network, each out-migrated VM andin-migrated VM being identified associated with a corresponding endpoint on said VLAN, said bridge further operable to: generate a VLANpacket by stripping off said fabric header from a corresponding switchedpacket received from said switched fabric, and adding an outer header,said outer header containing said destination IP field set to said firstIP address, said outer header also identifying a corresponding virtualnetwork identifier (VNI) indicating a segment of said VLAN; and sendsaid VLAN packet on said IP based network, wherein said bridge is alsoaddressable by an IP address, said outer header of VLAN packetcontaining said IP address of said bridge in a source IP address fieldwhen said VLAN packet is transported to said first VM migrated to saidfirst external system.
 5. The first appliance of claim 4, wherein saidbridge further comprises: a VTEP (virtual tunnel end point) tablemapping the MAC address of said first VM in said first appliance to saidfirst IP address in said first external system, and said VNI; and anetwork manager to generate said VLAN packet using said VTEP table andto send said VLAN packet on said IP based network.
 6. The firstappliance of claim 5, wherein a third VM from said first external systemis in-migrated to said first appliance and assigned a local MID (machineidentifier), wherein said bridge further comprises: a bridge tablemapping the MAC address of said third VM to said local MID in said firstappliance, to an IP address of said third VM in said external system,and to said VNI, wherein said VTEP table contains additional entries,with each entry including a mapping of the remote VM in said firstexternal system to said first IP address.
 7. The first appliance ofclaim 6, wherein said switched fabric and said first appliance are inaccordance with InfiniBand (IB) technology, and said VLAN is inaccordance with VXLAN technology.
 8. The first appliance of claim 7,wherein said first external system is coupled to said IP based networkby a broadcast medium based LAN, wherein said LAN uses said broadcastmedium to send said MAC packets to said first VM.
 9. The first applianceof claim 7, wherein said first external system is also in accordancewith said IB technology, wherein said VLAN packet is processed at saidfirst external system to generate a second switched packet that isswitched by a second switched fabric in said first external system toforward said MAC packet to said first VM.
 10. A method of supportingvirtual machines (VMs) in a first computing system, said methodcomprising: hosting a first set of VMs on computing node contained insaid first computing system; identifying a condition requiring migrationof a first VM of said first set of VMs to outside of said firstcomputing system; determining an external system suitable for hostingsaid first VM, said external system capable of being implemented aseither a switched fabric based system or a non-switched-fabric basedsystem; migrating said first VM to said switched fabric based externalsystem, if said external system is implemented as said switched fabricbased system; and migrating said first VM to said non-switched-fabricbased external system, if said external system is implemented as saidnon-switched-fabric based system, wherein said first computing system isalso based on switched fabric technology.
 11. The method of claim 10,further comprising: generating MAC packets destined for forwarding tosaid first VM by a second VM of said first set of VMs, wherein each MACpacket is encapsulated by a fabric header to form a switched packet;switching each switched packet based on said fabric header, wherein saidswitching delivers each switched packet, including a first switchedpacket, to a bridge provided within said first computing system;generating an IP packet by stripping off said fabric header from saidfirst switched packet, and adding an IP header, said IP headercontaining a destination field set to an external IP address, saidexternal IP address being that of said switched fabric based externalsystem, if said external system is implemented as said switched fabricbased system, said external IP address being that of saidnon-switched-fabric based external system, if said external system isimplemented as said non-switched-fabric based system; and sending saidIP packet on an IP based network, wherein said generating and saidsending are performed by said bridge.
 12. The method of claim 11,wherein said bridge is also addressable by an IP address, said IP packetcontaining said IP address of said bridge in a source IP address fieldwhen said IP packet is sent on said IP based network.
 13. The method ofclaim 12, wherein said bridge is in communication with said externalsystem according to VXLAN, said method further comprising: maintaining aVTEP (virtual tunnel end point) table containing a mapping of a MACaddress of said first VM in said first computing system to said externalIP address, and also a VNI (virtual network identifier), said VNIrepresenting a segment according to said VXLAN; and generating said IPpacket as a VXLAN packet using said VTEP table, wherein said sendingsends said VXLAN packet as said IP packet on said IP based network. 14.The method of claim 13, further comprising: in-migrating a third VM fromsaid external system to said first computing system, wherein said thirdVM is assigned a local machine identifier (MID) in said first computingsystem; maintaining a bridge table containing a mapping of the MACaddress of said third VM to said local MID, and to said VNI; receiving asecond packet from a fourth VM in said external system, said secondpacket containing said MAC address of said third VM and said VNI;determining said local MID of said third VM, by examining said bridgetable; and encapsulating a MAC portion of said second packet with saiddetermined local MID such that said switching operates to forward theresulting encapsulated packet to said third VM.
 15. The method of claim14, wherein switching is based on a forwarding table containing aplurality of entries, with each entry indicating a port at which a VMwith a corresponding MID is connected, said method further comprising:mapping, in said third VM, the MIDs of all VMs in said external systemto an MID of said bridge, wherein said forwarding table contains anentry mapping the MID of said bridge to a port on which the bridge isdirectly connected in said first computing system; sending, from saidthird VM, a third MAC frame to a fourth VM hosted in said externalsystem, said third MAC frame being encapsulated by a fabric headercontaining said MID of said bridge indicating that said bridge is thedestination of said third MAC frame, wherein said bridge, upon receivingsaid third MAC frame, forwards said third MAC frame to said fourth VM onsaid external system based on information contained in said bridge tableand said VTEP table.
 16. A non-transitory machine readable mediumstoring one or more sequences of instructions for enabling a firstcomputing system to support virtual machines (VMs), wherein execution ofsaid one or more instructions by one or more processors contained insaid first computing system enables said first computing system toperform the actions of: hosting a first set of VMs on computing nodecontained in said first computing system; identifying a conditionrequiring migration of a first VM of said first set of VMs to outside ofsaid first computing system; determining an external system suitable forhosting said first VM, said external system capable of being implementedas either a switched fabric based system or a non-switched-fabric basedsystem; migrating said first VM to said switched fabric based externalsystem, if said external system is implemented as said switched fabricbased system; and migrating said first VM to said non-switched-fabricbased external system, if said external system is implemented as saidnon-switched-fabric based system, wherein said first computing system isalso based on switched fabric technology.
 17. The non-transitory machinereadable medium of claim 16, further comprising: generating MAC packetsdestined for forwarding to said first VM by a second VM of said firstset of VMs, wherein each MAC packet is encapsulated by a fabric headerto form a switched packet; switching each switched packet based on saidfabric header, wherein said switching delivers each switched packet,including a first switched packet, to a bridge provided within saidfirst computing system; generating an IP packet by stripping off saidfabric header from said first switched packet, and adding an IP header,said IP header containing a destination field set to an external IPaddress, said external IP address being that of said switched fabricbased external system, if said external system is implemented as saidswitched fabric based system, said external IP address being that ofsaid non-switched-fabric based external system, if said external systemis implemented as said non-switched-fabric based system; and sendingsaid IP packet on an IP based network, wherein said generating and saidsending are performed by said bridge.
 18. The non-transitory machinereadable medium of claim 17, wherein said bridge is also addressable byan IP address, said IP packet containing said IP address of said bridgein a source IP address field when said IP packet is sent on said IPbased network.
 19. The non-transitory machine readable medium of claim18, wherein said bridge is in communication with said external systemaccording to VXLAN, said actions further comprising: maintaining a VTEP(virtual tunnel end point) table containing a mapping of a MAC addressof said first VM in said first computing system to said external IPaddress, and also a VNI (virtual network identifier), said VNIrepresenting a segment according to said VXLAN; and generating said IPpacket as a VXLAN packet using said VTEP table, wherein said sendingsends said VXLAN packet as said IP packet on said IP based network. 20.The non-transitory machine readable medium of claim 19, furthercomprising: in-migrating a third VM from said external system to saidfirst computing system, wherein said third VM is assigned a localmachine identifier (MID) in said first computing system; maintaining abridge table containing a mapping of the MAC address of said third VM tosaid local MID, and to said VNI; receiving a second packet from a fourthVM in said external system, said second packet containing said MACaddress of said third VM and said VNI; determining said local MID ofsaid third VM, by examining said bridge table; and encapsulating a MACportion of said second packet with said determined local MID such thatsaid switching operates to forward the resulting encapsulated packet tosaid third VM.